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An Evaluation of a State-Wide Mastery Learning Project 
in Civil Exgineering Technology: Was It Worth It?1 


Gregg M. Strasler 
D. Kent Sharples 
James D. Smith, dr. 


Introduction 


The purpose of the National Science Foundation (NSF) grant was to develop an indi- 
vidually-paced instructional format for a state-wide Civil Engineering Technology 
curriculum in South Carolina. In the past three years, approximately 160 compe- 
tencies were identified for eleven core courses in the civil engineering program. 
The basic format for each course was a combination of written modules forming a 
linear progression through a series of related objectives. In some cases the 
written material was broadened by a series of media presentations for students 
having difficulty understanding particular concepts presented in the modules. 


In an effort to determine the effectiveness of the materials developed, a state- 
wide evaluation was conducted during the fall, winter, and spring quarters of the 
1975-1976 academic year. The evaluation basically utilized a pretest/posttest 
design to compare students taught by an individually-paced instructional format to 
students taught by a more conventional lecture-based approach. 


Definitions of Experimental and Control Strategies 


As mentioned in the Introduction, two instructional strategies were utilized in the 
evaluation: the individually-paced method and the lecture based method. These 
teaching strategies wil! be referred to as the experimental and control strategies, 
respectively. A brief introduction to these instructional formats is outlined below. 


(1) Individually-Paced Method of Instruction (Experimental) 


The individually-paced curriculum is similar to Keller's Personalized System of 
Instruction (PSI) in that it focuses on individual learning based on small units 
of instruction. These units of instruction are in the form of modules which are 
usually completed by the student in a week's time or less. Like PSI, this 
method of instruction is individual-oriented rather than group-oriented. How- 
ever, there are essential differences between the Keller plan and the indi- 
vidually-paced curriculum. Some of these differences are: (1) students move 
through the modules (learning units) at a prespecified pace; (2) a demonstration 
of mastery (80%-95%) is required at the end of each module before the student 

is permitted to continue into the next module (however, these module posttests 
do not count toward the student's final grade); (3) slide-tape presentations 

and other alternative (non-reading) learning methods are available for indi- 
vidual use within each module; and, (4) the student's final grade is determined 
primarily by his performance on a final exam (instructors had the option of 
using a mid-term as well). 


T Consulting services were provided by Dr. Garrett K. Mandeville and Dr. Lorin W. 
Anderson of the University of South Carolina. The project staff, however, assumes 
full responsibility for the final interpretation of the results. 
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(2) Lecture-Based Instructional Method (Control) 


One of the most common types of instructional strategies, lecture-based instruc- 
tion, is basically textbook oriented. The instructor plays the vital role of 
lecturing on key ideas and concepts specified by the course objectives. The 
lecture-based method is a group-based strategy where the time allowed for 
learning is relatively fixed. The course is divided into larger learning units 
and the student is given a test at the end of each unit which counts toward his 
final grade. Therefore, the student's grade is determined by his performance 

on the unit tests (numbering between two and four in most cases) as well as his 
performance on the final exam. 


Although these instructional strategies were purposely contrasted, the evaluation 
procedures should not favor one strategy over the other. All of the testing instru- 
ments used to measure student performance were based on prespecified course objec- 
tives developed by the instructors on the NSF grant. With the exception of one out- 
of-state instructor, all of the instructors were quite familiar with the objectives 
of each course.! In viewing the distinctive features of the instructional strategies, 
all instructors had copies of the modules and objectives contained with each course. 
Aside from the characteristics noted above, the instructional stratecies differed 

On one important point: the individually-paced method utilized the instructor as a 
manager of student learning, whereas the lecture-based method enabled the instructor 
to become the main dispenser of information. The objectives were already stated and 
in written format (i.e., in modular form) for the individually-paced approach, 
whereas the dissemination of objectives and written format was primarily the respon- 
sibility of the instructor in the lecture-based approach. In essence, both instruc- 
tional stragegies were designed to emphasize student learning with respect to the 
same course objectives. A basic difference between the instructional strategies 

was the way in which these objectives were approached. 


Data Source 


The sample consisted of 250 male students enrolled in two-year, post high school 
engineering technology programs. Of this total, 220 students were located in seven 
technical education colleges throughout the state of South Carolina, and 30 students 
were located in one community college in North Carolina. In all of the two-year 
institutions, the majority of students ranged from eighteen to twenty-two years of 
age (65%), were non-veterans (70%), and were enrolled in a civil engineering program 
(65%). Those students not enrolled in civil engineering programs (35%) were 
enrolled in related engineering fields (e.g., engineering technology, architecture, 
building construction, and engineering graphics). 


Assignment to Groups 


Instructors were allowed to choose the instructional format (individually-paced or 
lecture-based method) they preferred. This procedure for assigning instructors to 
treatment groups presents a possible methodological problem: a self-selection of 


I The out-of-state instructor taught civil engineering courses at a two-year community 
college in North Carolina. This community collece was used in attenpting to answer 
Problem Statement 4. 
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teachers into treatment groups that is probably not random. While random assignment 
of teachers to treatment groups may be construed as a methodological ideal, true 
randomization and experimental design in a school setting are nearly impossible to 
attain from a practical viewpoint. Given the diverse nature of the two instructional 
formats, the instructors would have likely chosen the teaching method with which 
they were most comfortable and presumably, most likely to be successful. In essence, 
allowing instructors to select teaching method was viewed as a strength, not a weak- 
ness, of the evaluation. 


Design of the Analysis 


A one-way analysis of variance design was the basic strategy for analyzing the results 
of the evaluation. The design includes the effect due to treatment (experimental vs. 
control groups). The model for the analysis was: 


Y=u+T+E, where 

Y = the dependent variable, 

u = the grand mean, 

T = the treatment effect, and 
E— = random error. 


The treatment effect was tested by comparing the mean square due to treatment against 
the mean square due to random error. 


Limitations of the Study 


The design of the analysis appears to utilize a rather simplistic model given the 
relative complexity of the evaluation problem statements. In addition to the effect 
due to treatment, a more appealing analysis would have been to include campus as a 
factor as well as classes nested in campus and subjects nested in classes. Inter- 
action effects such as treatment by campus would also have been appropriate in a more 
sophisticated analysis of variance design. 


The main deterrent to using a more sophisticated design was the limited number of 
classes evaluated in each course. In some instances, only a single class in one 
course was evaluated; in other cases, there were four or five classes evaluated per 
course. The lack of appropriate numbers of classes was primarily due to the fact 
that course materials were not sufficiently developed in time to evaluate them. In 
some cases, classes were rescheduled or cancelled; in other cases, classes were 
eliminated because instructors were not implementing the instructional strategies as 
they were operationally defined. The small number of classes per course, therefore, 
eliminated the possibility of incorporating analysis of variance design models using 
such terms as classes nested in campus and treatment by classes nested in campus 
interaction. The evaluation might have been more profitable had fewer courses with 
a larger number of classes been investigated. Nevertheless, all of the courses 
sufficiently developed during the evaluation were analyzed. 


Results 
This section deals primarily with the analysis of the six evaluation problem state- 


ments cited in the Introduction. Appropriate tables are placed at the end of each 
problem statement to preserve reading continuity. 
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Problem Statement 1 


Do students learn more under the individually-paced curriculum than under more conven- 
tional teaching methods? 


In an effort to resolve this issue, cognitive achievement in ten Civil Engireering 
Technology courses was analyzed with respect to two operationally defined instruc- 
tional strategies: an individually-paced method and a lecture-based method. For 

each particular course, a fifty-item summative test was administered to students 

both as a pretest and as a posttest. Each summative test was constructed from a 1 
crude table of specifications tapping the content covered and behavior level required. 
In addition to the summative tests, students were also required to take three subtests 
(Reading Skills, Mathematical Usage and Nonverbal Reasoning) of the Career Planning 
Profile (CPP), 


The CPP subtests and the,Summative Pretest were administered in each course to assess 
student entry behaviors.“ Results of these measures may be found in Tables 1 and 2. 
Table 1 displays the Pearson product-moment correlations of the CPP subtests and the 
Summative Pretest with the Summative Posttest by instructional method (individual ly- 
paced vs. lecture-based) in each of the ten Civil Engineering Technology courses. In 
essence, Table 1 shows the relationship between student entry behaviors and student 
exit behaviors. The results portrayed in Table 1 were sporadic: the correlations 
ranged from -0.87 to +0.70. With approximately half of all the correlation coeffi- 
cients falling within a range of +0.30, relatively few significant correlations were 
found. These sporadic results may have been due, in part, to the low numbers of 
students in both experimental and control groups within particular courses. 


Despite the sporadic results observed in Table 1, an attempt was made to see whether 
student entering behaviors in particular courses were significantly different with 
respect to treatment (e.g., experimental vs. control). With treatment as the main 
effect, one-way analyses of variance were performed using the Summative Pretest and 
two of the CPP subtests (Reading Skills and Mathematical Usage) as dependent variables 


TA rating scale was devised to determine the essential course content needed to be 
covered and completed by the "average" student within an eleven week quarter. 
Instructors were asked to rate each of the course objectives with one of three 
alternatives: "crucial," "“useful>but not essential," or "not important." The 
behavior leve? of an item was classified as either "knowledge" or "comprehension." 
Knowledge items were stated in a form similar to that in which the content was 
taught. Comprehension items were stated in forms at higher levels than knowledge 
items; that is, comprehension items included the levels of comprehension, appli- 
cation, and analysis described in Bloom's taxonomy (Bloom, B.S. Taxonom of Edu- 
cational Objectives: Cognitive Domain, David McKay Company, Inc. New York. 1955). 


ip] 


In this context, student entry behaviors are defined as the aptitude and/or achieve- 
ment measures students possess prior to instruction (or, in this instance, prior 

to taking a particular course). Student exit behaviors, on the other hand, are 
defined as those measures students possess after receiving instruction. Student 


entry behaviors were assessed bacause students were not randomly assigned to 
experimental and control classes. 
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in each of the ten Civil Engineering Technology courses./ Results in Table 2 indicate 
that there were no statistically significant differences found (with the exception of 
the math subtest in Structural Steel) between the experimental and control groups on 
the reading and math subtests. On the other hand, there were statistically signifi- 
cant differences observed between the two groups on the Summative Pretests in four of 
the five courses analyzed. Statistically significant differences favoring the control 
group were found in the following three courses: Highway Design and Construction, 
Surveying I and Structural Steel.2 In summary, the student entering behaviors mea- 
sured did not appear to favor the experimental group. In terms of the reading and 
math subtests of the CPP, the two groups were rclatively equal; in terms of the 
Summative Pretest, the control group maintained a slight advantage. 


Student’ exit behaviors, as measured by the Summative Posttest in each course, were 
analyzed in the same manner as the student entry behaviors.3 One-way analysis of 
variance results using treatment (experimental vs. control) as the main effect may be 
observed in Table 2. Due to an absence of an experimental groyP or a control group, 
only six of the ten courses could be analyzed in this fashion.* Statistically signifi- 
cant differences favoring the experimental group were found in three of the six courses: 
Highway Design and Construction, Structural Steel and Structural Concrete. In the case 
of the first two courses mentioned, statistically significant differences favoring the 
control group were found on the Summative Pretest; in the case of Structural Concrete, 
statistically significant differences favored the experimental group on the Summative 
Pretest. Although not statistically significant, differences in favor of the experi- 


mental group were also found in the Surveying I and Surveying II Summative Posttests.° 


T The CPP Nonverbal Reasoning subtest was not included in the analysis for two reasons. 
First of all, this subtest was negatively correlated to more of the Summative Post- 
tests than were the reading and math measures. In addition, only one of the corre- 
lations was found to be statistically significant. The reader should also note 
that comparisons between experimental and control groups could only be made in six 
of the ten courses using the reading or math subtests as dependent variables and 
in only five courses using the Summative Pretest as the dependent variable. 


2 The reader should note that the mean of the experimental group in Structural Steel 
Xe = 8.2 <Xc = 13.7) and the mean of the control group in Structural Concrete 
X¢ = 7.4 < Xe = 20.6) were quite low. By chance alone, a student should be able 

to quess correctly on twelve to thirteen items on a fifty item test with four 
alternatives per item. The low means noted possibly indicate that students were 
ret really trying on these particular Summative Pretests. 


w 


The reader may question analyzing the Summative Posttests without using some or 
all of the student entry behaviors as covariates. Although one of the uses of 
analysis of covariance is to increase precision, the results in Table 1 indicate 
that neither the CPP subtests nor the Summative Pretest were correlated consis- 
tently high enough with the Summative Posttest to be used as covariates. 


> 


Four of the courses contained only an experimental or control group. Of these fcur 
courses, three were individually-paced. The experimental groups in Statics, Cost 
Estimating and Soil Mechanics made Summative Pretest/Summative Posttest mean gains 
of 16.1, 11.6 and 11.0 raw score points. In the one course using the lecture- 
based method (Strength of Materials), the Summative Pretest/Summative Posttest 

mean gain of the control group was 3.4 raw score points. 


on 


In the case of Surveying I, statistically significant differences in favor of the 
control group were found on the Summative Pretest. No significant differences 
were observed on the Surveying II Summative Pretest. 
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Differences favoring the control group were found in only one course, Hydraulics, 
although this ditference was not statistically significant. In terms of student entry 
and exit behaviors, then, the individually-paced students appear to have made more 
cognitive gains than those students taught by the lectured-based method of instruction. 


As an interesting sidelight, the results of the Summative Pretest and Summative Post- 
test are also presented in Table 3. This table includes the means, standard devi- 
ations and raw score ranges of the experimental and control groups in the ten Civil 
Engineering Technology courses evaluated. Based on Bloom's (i976) thinking concerning 
individual differences, one would expect that the variation of scores on the Summative 
Posttests would be less for individually-paced students (where "mastery" was required) 
than for lecture-based students.! With the exception of Surveying II, Hydraulics, and 
Structural Concrete, however, just the opposite was true: the standard deviations and 
raw score ranges of the Summative Posttests were slightly larger in individual ly- 
paced courses than in lecture-based courses. In essence, there appeared to be as much 
variability among individually-paced students as there was with lecture-based students 
in terms of student exit behaviors. This increased variability may be, in part, 
attributed to the emphasis placed on individual learning rather than on group learning. 


T Bloom, B.S. Human Characteristics and School Learning. New York: McGraw-Hill, 1976. 
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Table 1, Pearson product-moment correlations of summative pretest, CPP subtests 


(reading, math, and nonverbal) with summative posttest in ten civil 
engineering technology courses. 


CPP CPP CPP | 
Reading Math Nonverbal 


Summative 
Pretest 


COURSE 


Oo 
— O 
Ww Ww 
w 
~— 


ee Ee 0.22 || 0.26 | 0.30 ||-0.23 
Construction (11 (8)| (23) (8)} (23) (8) 
2 1 0.20 


Surveying II 0. 


[al tal kal al“ 
(15) | (12) (15) (12) 
a eall - Peal ~ [én 

(40) (40) (39) 
Be Ks 
ee (26) (12)] (26) (12) (26) 
a re 
(28)| (32) (28)| (32) (28) 31) 
[Pol = [al = Pal 


Statics 


~ wu 
—_ 
n 
~~ 
a 
[oe] 
~~ 


Oo 
nae 
[oe) 

* 

* 


2 
19) 
8 |-0.42* 
(39) 


Surveying I 


Structural 0. 
Steel 


~o 
w 
— 
~~ 


Strength of 
Materials 


Oo 
Se ey 
— © 
w 
~— 

So 

e 
-——m W 


— a 
™~“N 
~ 


Hydraulics 


6) 
3a aed a 
12 12 
zl — 
10 10 


o 

ar 

wo |— _ 
> > 


Cost 
Estimating 


-0. ne 0.59* | 0.27 
Concrete (21 13 13 


Mechanics 
Yc = product-moment correlation in control group 


Structural -0,53**} 0.35 


le 
8 


“(10 


NOTE: 


Ye = product-moment correlation in experimental group 


Asterisks indicate degree of significance: * (P < .05), ** (P< .01) 


Table 2. Analysis of variance results for summative, posttests, pretests and CPP subtests (reading and math) in ten 
civil engineering technology | 
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Table 3. Descriptive statistics for summative pretests and summative posttests 
in ten civil engineering technology courses. 


ummative Pretest | Summative Posttest 
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NOTE: 
The "Raw Score Range" denotes the lowest and highest score (number of items 


correct) recorded on each fifty (50) item summative pretest and summative 
posttest. 
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Problem Statement 2 


An_individually-paced curriculum may produce a very different grade distribution than 
that_seen in a conventional course, as many more students may receive A's and B's in 
an_individualTy-paced curriculum. Are these higher grades justified? 

Instructors were required to make available the final letter grades they assigned to 
their students in the spring quarter. Structural Concrete was the only course 
analyzed because of the limited number of classes in the other spring courses. In 
Structural Concrete, there were two control classes of eight and twelve students, and 
one experimental class of fourteen students. The results observed in this section 


should be interpreted with caution due to the fact that only one ccurse with a limited 
number of classes was analyzed. 


A one-way analysis of variance was used to analyze the results. Table 4 indicates 
there were no significant differences between experimental and control groups with 
respect to final ane. assigned. In fact, final grades were slightly higher for the 
control classes (Xc = 3.3) than for the experimental class (Xp = 3.2). 


The grade distribution for the experimental and control classes reveals the following 
information: 


Grade Distribution Percentages 


(N = 14) 


In both the experimental and control classes, the majority of students received "A's" 
or "B's" (86% in the experimental group vs. 80% in the control group). A chi-square 
value of 0.19 was obtained by comparing the proportion of "A's" and "B's" to the 
proportion of "C's" and "D's" in the experimental and control groups. With one degree 
of freedom, this chi-square value was not significant. In the case of Structural 
Concrete, there appears to be no dependent relationship between grades assigned and 
instructional method used. 


The last portion of the problem statement deals with the question of whether higher 
grades in an individually-paced curriculum are justified. This question would have 
been applicable if there had been differences observed in the means of the final grades 
assigned in the experimental and control qroups; however, there were no significant 
differences observed. If differences in grades had been observed, the problem of 
justification might have veen demonstrated by using analysis of covariance (ANCOVA) 
with the Summative Posttest as a covariate and, hopefully, achieving non-significance 
(with the assumption that Summative Posttest differences observed would be "adjusted 
for" in the ANCOVA). 
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Table 4. Analysis of variance results for final grades assigned in Structural 
Concrete, 


Final Grades Assigned 


Structural 3a Koes 0.02 N.S. 
Concrete (20) (14) (1,31) 


NOTE: 


Xc = mean of control group 
Xe = mean of experimental group 


Grades were determined by the following scoring scheme: A=4, B=3, C=2, D=1, F=0 


Problem Statement 3 


Do individually-paced students exhibit a significantly different long-term retention 
of facts and concepts than students taught 7 a_Tecture method? 

Ten weeks after a Summative Posttest in Structural Steel had been administered, a 
twenty-item Retention Test was given to twenty-five students taught by the lecture- 
based method and to twelve students taught by the individually-paced method. The 


Retention Test was considerably shorter than the Summative Posttest so that it could 
be administered in a one hour time period. 


From the results observed in Problem Statement 1, students in individually-paced 
classes significantly outperformed students in lecture-based classes on the Structural 
Steel Summative Posttest. This same level of significance is also observed in Table 
5, where the means of the Sunima ti ye Posttest in Structural Steel are based on twenty 
items rather than on fifty items. 


I The Retention Test was merely a shortened version of the Structural Steel Summative 
Posttest. Twenty items were selected by an instructor who had taught the Structural 
Steel course by the lecture-based method. As in the Summative Posttest, 40% of the 
retention items were classified as knowledge items and 60% were classified as compre- 
hension items. 


2 Two points need to be emphasized. First of all, only the twenty "conmon" items 
included on both the Retention Test and the Summative Posttest in Structural Steel 
were used in the analysis. Secondly, the reader will note there were fewer students 
(N = 38) receiving the Retention Test than there were students (N = 6&) receiving 
the Summative Posttest. The loss of students was primarily due to administrative 


difficulties. 
14 


iz 


One-way analysis of variance results using treatment (experimental vs. control) as 

the main effect may be observed in Table 5. Although the results were not signifi- 
cantly different (P = 0.07), the experimental group scored higher than the control 

group on the Retention Test (XE = 9.9 > Xc = 7.9); however, the question of initial 
performance on the Summative Posttest must be resolved. 


Two methods of analysis were used to investigate "retention" in terms of the initial 
performance on the Summative Posttest. The first method involved analysis of the 
experimental and control groups in terms of "pure retention." A "Difference Score," 
calculated by subtracting the mean of the Retention Test from the mean cf the Sum- 
mative Posttest, is presented in Table 5. In terms of "pure retention," the experi- 
mental students appear to have lost more than the control students in the absolute 
sense (XE = 4.2 > Xc = 1.7) although the experimental students scored higher overall 
on the Retention Test. The second method regressed the Retention Test scores onto 
the Summative Posttest scores for the control group. Predicticn of the Retention 
Test score for the experimental group was obtained and compared to the actual Reten- 
tion Test score.! A one-tailed t-test (t = 0.57) revealed that the difference 
between the actual value and the predicted value of the Retension Test score for the 
experimental group was not significantly different from zero. 


One final method was used to analyze the Retention Test results. Table 6 reveals chi- 


square results for each of the Structural Steel retention items. In accordance with 
the two methods of analysis used above, initial differences in Summative Posttest 


performance were taken into account by analyzing only those students who got a specific 


item correct on the Summative Posttest. A 2 X 2 contingency table of the following 
form appeared appropriate in analyzing the twenty Retention Test items: 


: correct 
Retention 


Test incorrect 
Ne Nc 


where Ne and Nc are the number of students in the experimental and control groups 
who answered the item correctly on the Summative Posttest. 


I The predicted Retention Test score of the experimental group was obtained from the 
following regression equation: Y = 3.57 + 4.18 (Posttest Score). The predicted 
Retention Test score of the experimental group (Y = 9.46) was then subtracted from 
the actual Retention Test Score (Xe = 9.92) observed in Table 5. A difference of 
0.46 was observed between the predicted and actual (observed) Retention Test score 
for the experimental group. 


2 The t-value was calculated as follows: 


0.46 
i tay 
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To observe whether item retention differed between experimental and control groups, 
a chi-square analysis was applied to the contingency tables outlined in Table 6. 
With the exception of the first two items, the values of chi-square were not found 
to be significant at the 0.05 level in testing for differences in the proportions of 
items correct on the retention test for the two instructional methods. In essence, 
there appears to be no dependenx relationship between the amount of item retention 
and the instructional method utilized. 


T Although the majority of chi-square values were not found to be significant, Table 
6 merits further investigation. The proportions of students answering retention 
items correctly appear to be slightly in favor of the experimental method. There 
were only three clear-cut cases in which the control students outperformed the 
experimental students (i.e., items 10, 13, and 15); on the other hand, there were 
nine items which favored the experimental students (i.e., items 1, 2, 3, 4, 5, 7, 
14, 18, and 20). There appeared to be no "direction" for the eight remaining 
items. 
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Table 5 . Analysis of variance results for summative posttests, retention tests, and difference scores in 


Structural Steel. 
Summative Posttest” Retention Test Difference Score** 


COURSE Xe P | Fe | Xe leary P ii X%c | Xe (| F(at)  P 


Structural 7.9 9.9 {3.44 N.S bh Tat i215 ,54 <.05 
(12) |(1,35) (25) | (12) | (1,35) 


Steel 


NOTE: 
Xe = mean of control group 
Xe = mean of experimental group 


* The means of the Summative Posttest in this table are based only on the 20 items used for the Retention Test in 
tructural Steel. 


- The Difference Score is calculated by subtracting the mean of the Retention Test from the mean of the Summative 
Posttest. 


tT 
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Table 6. Chi-square results of structural steel retention items. 


Item Soar Correct Incorrect 2 
Classification Responses | Responses | X (1df)  ? 
zs Knowledge 2 


3 


Treatment Group 


Comprehension 


Control J 


Experimenta | 11 peek as 


ag mites Sear 
conprehension 
| Control 


Comprehension contro 


Comprehension 


Experimental 


=e 
as 
5 
; Comprehension 
11 
13 
Cd 
15 
16 
17 
18 
19 


Control —+-— . 5. 
Se = 
[Knowledge | “rxperinantsT—[ 1 
Comprehension 
[eomeesion fet — 
| 8 | comretension | piper mentaT—[—— + —J 
14 Comprehension 
Control 


Comprehension 


| 17 | knowtedge g 


3 ae ea 
an) 

Knowledge y experimental | 1 _| 
ae el 

ype a ee 
Ptontrol 

| 18 | Knowledge P_ Experimental [8 
oa iene! 

7 Lo el 

aaa cna 

as ee 


1 
’ 
Knowledge ae eee 
9 U 
Knowledge Experimenta 


insufficient 


20 data for y? 


NOTE: 


The number of total responses (correct responses and incorrect responses) within 
treatment group varies from item to item. This is due to the fact that only students 
correctly answering items on the summative posttest were analyzed on the respective 
retention test items. 
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Problem Statement 4 


Can the individually-paced curriculum materials developed by a given instructor be used 
in another institution with roughly equivalent results? That is, are well-written 
individually-paced curriculum courses relatively independent of the actual instructor, 
provided the FAS trac tor using them understands how the individual ly-paced curriculum 


WOrKS § 


An attempt was made to answer the above problem in part by using some of the individu- 
ally-paced materials in an institution outside the state of South Carolina. Sandhills 
Community College, a two-year institution located in North Carolina, agreed to parti- 
cipate in the evaluation by using the individually-paced instructional method. Two 
experimental classes, one in Highway Design and Construction and another in Surveying 
II, were taught during the spring quarter in Sandhill's Civil Engineering Technology 
program. 


The general characteristics of the engineering technology students in Sandhills Commu- 
nity College compared favorably to the technology students enrolled in the South 
Carolina schools. The majority of Sandhill's engineering technology students ranged in 
age from eighteen to twenty-two years of age (65%), were non-veterans (75%), and were 
enrolled in civil engineering programs (65%). Those students not enrolled in civil 
engineering programs (35°) were enrolled in architectural technology, a related engi- 
neering field. 


The students enrolled in the two experimental classes at Sandhills were also compared 
jin terms of student entering behaviors to the South Carolina students enrolled in the 
experimental classes in Highway Design and Construction and Surveying II. Table 7 
indicates there were no significant differences observed on the raw scores of the CPP 
reading and math subtests between the two experimental groups (South Carolina schools 
vs. Sandhills Community College) in either the Highway Design and Construction or the 
Surveying II courses. There appeared to be no distinct differences between the 
experimental groups in terms of general characteristics and student entering behaviors. 


The two experimental groups were compared on the Summative Pretests and Summative Post- 
tests administered in the two courses. One-way analysis of variance results using 
treatment (experimental: South Carolina vs. Sandhills) as the main effect may be 
observed in Table 7. The results concerning the Highway Design and Construction course 
are confounded by the fact that five items (items 51-55) were missing on the Sum- 
mative Pretest and Summative Posttest administered to the BAEN TEN ta group in North 
Carolina; therefore, interpretation of these results is suspect. 


With respect to the Surveying II course, Table 7 reveals there were no significant 
differences between the two experimental groups on the summative tests. Both groups 
appeared to do equally well on the Summative Pretest (Xp] = 23.9 = Xeo = 20.9) and 
the Summative Posttest (Xpz = 39.1 = Xg2 = 38.1) in Surveying II. A eheugh the data 
is somewhat limited, the results from the Surveying II course tend to support the 
idea that the individually-paced curriculum could be used effectively elsewhere. 


I Although there were five missing test items for the North Carolina students, they 
significantly outperformed the South Carolina students on the Summative Pretest; 
however, the florth Carolina students were sianificantly behind the South Carolina 
students in terms of the Summative Posttest. 
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Table 7, Analysis of variance results for summative posttests, pretests and CPP subtests (reading and math) in two 
civil engineering technology courses. 


one Summative Posttest Sumiative Pretest CPP Reading 4 CPP Math 
COURSE Xe | %e2 “|Fcaey P || Xe | Xe2 | Feary | Xer_ | Hee | Xeo_| Flee) Yer | Xee_|Ftary 


Hichway Bester ® | 95.8 — <.05)| 18.7 23-0 |12.1 <.01/) 31.2 28.8 11 60 
and Construction | (33) | (11) re (1 20) (33) | (41) | (1,49) | (23) | (20) | (2,29 (10) | (1,29) 


1 : 
| ( 
38.9 N.S.]] 23.9 0.71 N.S.|| 29.8 realli 46 me 
Surveying II ret a 00) 1s (10) (1, 16) (12) | (11) | (1,22) 


Xey = mean of experimental group in South Carolina 
2 2 mean of experimental group in North Carolina (Sandhills Community College) 


* Five items were missing on the summative pretest and posttest administered to the experimental group 
in North Carolina which may account for some of the differences noted. 


it 
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Problem Statement 5 


The majority of people who have taught self-paced courses have experienced the "procras- 
tination” problem evidenced by a significant number of students who progress through 

a course at a rate much slower than average. What are the various causes of procras- 
tination and how may a course be designed to minimize the problem? 


Two instruments were used in attempting to answer the problem statement: a Course 
Questionnaire and an Instructor Questionnaire. 


As noted in the problem statement, a procrastinator is defined as a student who pro- 
gresses through a course at a rate much slower than average. In an effort to identify 
procrastinators in both individually-paced and lecture-based groups, the definition 

of procrastinator was expanded to include any student who dropped a course, received 
an incomplete, or failed a course. 


Of the total number of students identified as procrastinators in the fall, winter, and 
spring quarters (N = 53, 58% responded to the Course Questionnaire. The general charac- 
teristics of those responding to the Course Questionnaire compared favorably to the 
total sample of 250 students: the majority of procrastinators ranged from eighteen to 
twenty-two years of age (62%), were non-veterans (70%), and were enrolled in a civil 
engineering program (75%). 


To observe whether the rate of procrastination differed between the two instructional 
methodologies employed, a chi-square analysis was applied to the contingency tables in 
Table 8. The values of chi-square were not found to be significant at the 0.05 level 
in testing for independence of procrastination and instructional method. A chi-square 
value of 1.36 was observed for the classification of procrastinators (e.g., students 
dropping a course, receiving an incomplete, or failing a course) and students passing 
a course versus instructional method. A chi-square value of 9.50 was observed for the 
classification of students dropping and students passing a course versus instructional 
method. Therefore, there appears to be no dependent relationship between the amount 
of procrastination and the type of instructional method utilized. There was no 
observed tendency for students to procrastinate more in individually-paced courses than 
in lecture-based courses. However, one should be reminded that the individually-paced 
approach was not totally self-paced; that is, students were given guidelines to move 
through the modules (learning units) at a prespecified pace. Also, students in the 
experimental group were recuired to take the final exam (Summative Posttest) at the 
end of the quarter and were sometimes required to take a midterm: therefore, the 
requirements of the individually-paced instructional method may have deterred procras- 
tination evident in many self-paced instructional approaches. 


A section entitled "Course Evaluation" in the Course Questionnaire was not utilized in 
the evaluation. This was due to the small number of procrastinator returns per indi- 
vidual course. However, an open-ended "Course Comments" section was content analyzed 
with respect to procrastinators in the.individually-paced courses. Procrastinators 
listed the three "major limitations" of the individually-paced format as being: 


(1) too many students per class with only one instructor; 
(2) too much reliance on modules (learning units) for learning; and, 
(3) students must be self-motivated. 


Proc'astinators listed the three most common methods for “improving instruction" in 
the individually-paced format as beina: 
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2) introduce one lecture per week for review purposes; and, 
3) prepare a schedule ("time table") for the completion of each module; enforce- 
ment of this schedule will help motivate students to complete a course. 


a decrease the instructor-student ratio (e.g., limit class size); 


The procrastinators in individually-paced courses appeared to be asking for more instruc- 
tor input as well as for a more rigidly defined course time-table. 


Instructors were asked on the Instructor Questionnaire to list the three major advan- 
tages and disadvantages of the particular instructional format (individually-paced or 
lecture-based) used. Listed below are the three major advantages and disadvantages for 
the two types of instruction. 


(1) Individually-Paced Method 
A. Advantages: 


1. Students may progress at a pace consistent with their abilities. 

2. A student may be recycled until acceptable mastery of individual 
learning objectives is attained. 

3. The student is primarily responsible for learning. 


B. Disadvantages: 


1. The instructor-student dialogue (interaction) is impaired. 

2. There are larger demands upon instructor time (dependent directly on 
the instructor-student ratio). 

3. Students have a tendency to procrastinate and fall behind schedule. 


(2) Lecture-Based Method 


A. Advantaqes: 
1. The instructor has control over the class aS a group. 
2. Classroom discussion and instructor-student interaction is enhanced. 
3. The instructor's time is effectively utilized. 


B. Disadvantages: 


1. The instructional method is geared toward the average anc below aver- 
age student; coverage of the materials is limited for the best students. 

2. A student must take regularly scheduled exams regardless of his degree 
of achievement. 

3. Many students do not become actively involved in the learning process. 


Generally speaking, instructors using the individually-paced stratecy stressed the impor- 
tance of that method in terms of student ability and motivation. Instructors utilizing 
the experimental method emphasized not only that learning was primarily the respon- 
sibility of the student, but also that learning should take place in the franiework of 

the student's ability. Viewing the disadvantages of the experimental method, the 
instructor-student ratio appears to be a critical feature. If the instructor-student 
ratio is reasonable, the disadvantages of the individually-paced method appear to be 
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minimal, 


Instructors using the lecture-based method appeared to emphasize the importance of the 
class as a group. The instructors using the control method stressed the importance of 
class control as well as class discussion. These instructors also felt their time was 
utilized more effectively. In observing the disadvantages cited by the instructors in 
the lecture-based method, there appears to be a concern for the student as an indi- 
vidual with respect to abilits’ level and motivation. These very characteristics were 
the ones emphasized in the advantages of the individually-paced instructional method. 


Table 8 . Contingency tables showing the relationship between procrastination 
and instructional method for 498 and 488 subjects, respectively. 


Number of Number of 
Procrasti- Students 


nators Passing Total 
Individually- 
Paced Method 361 
Lecture- 
Based Method 137 
Total 53 445 498 


X" (af) = 1.36, N.S. 


I In speaking with the instructors employing the individually-paced instructional inethod, 
the consensus for a workable ratio appeared to be one instructor for ten students or 


less. 
25 
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Table 8 . Continued 


Number of Number of 
Students Students 


Dropping Passing Total 
Individual ly- 
Paced Method 33 319 352 
Lecture- 
Based Method 10 126 136 
Total 43 445 488 


x” (1df) = 0.50, N.S. 


NOTE: The frequencies of incompletes and failures noted below were not high 
enough to perform chi-square analysis. 


Drops Incompletes Failures Passing Total 

Individual ly- 361 
Paced Method 

Lecture- 137 
Based lethod 

Total 43 7 3 445 age 
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Problem Statement 6 


Do students under an individually-paced curriculum display a more positive affect 
toward subject matter than students under more conventional teaching methods? 

A fifty item Likert scale (Course Evaluation) was administered to all students at the 
completion of each course. The Course Evaluation comprised eight subscales which were 


logically validated. The Course Evaluation was further vaiidated empirically through 
the use of a factor analysis technique. 


The technique of principal components analysis was employed using all subjects' responses 
(both experimental and orrte). experimental subjects' responses, and contro! subjects’ 
responses. Results of the factor analysis on the Course Evaluation indicated fifteen 
major rotated factors (a varimax rotation), each possessing eigen values greater than 

one (for all subjects, experimental subjects only, and control subjects only). Only 
items "loading" greater than or equal to 0.50 were considered in the interpretation of 
the factors. 


Although fifteen factors were extracted, only four factors (see Table 9) were used in 
the analysis. These factors were chosen because they fulfilled the following criteria: 


(1) Each factor had relatively few high lcadings of variables (items) with the 
rest of the loadings very close to zero. 

(2) The phraseology of each item was not biased toward the instructional method 
used in a course. 

(3) The variables (items) loaded significantly (0.50 or greater) on both instruc- 
tional methods (experimental and control). 

(4) Each factor was relevant to the problem statement. 


The Course Evaluation items corresponding to the four factor structures were labeled as 
follows: 


(1) Factor I.....Course Workload and Grading. 

‘2 Factor II....Instructor Effectiveness. 

3) Factor III...Attitude Toward Course and Instructional Method. 
(4) Factor IV....Relevancy of Course and Instructional Method. 


All of the items in each of the four subscales were dichotomously scored. An item 
stated in a positive manner would be scored as "1" if a response to the item was 
"strongly agree" or "agree." An item stated in a negative manner would be scored as 
"1" if a response was "strongly disagree" or "disagree." Therefore, a high point total 
on a subscale would indicate a positive reaction to that subscale and a low point total 
would indicate a negative reaction. One-way analysis of variance results using treat- 
ment (experimental vs. control) as the main effect may be observed in Table 10. 


In general, the lecture-based method was rated more positively than the individual ly- 
paced method with respect to the four subscales described. In particular, the control 
group scored significantly higher in six of eight courses on both the "Instructor 
Effectiveness" and the "Attitude Toward Course and Instructional Method" subscales. In 
four of eight courses the control group scored significantly higher on the "Coursework 
and Grading" subscale, and, in two of eight courses, the control group scored signifi- 
cantly higher on the subscale entitled "Relevancy of Course and Instructional] Method." 
The remaining courses showed no significant differences between experimental and contro] 
groups on the four subscales. The results appear to indicate that students under an 
individually-paced curriculum do not display a more "positive affect" toward subject 
matter than students under more conventional teaching methods. 
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Table 9 . Course evaluation items loading 0,50 ov greater for both experimental 
and control] students. 


Factor I (Course Workload and Grading): 


Factor Loading 
0.79 


0.65 
0.62 
0.61 
0.60 


0.55 
0.52 


Items 


(11) 


(46) 
(49) 
16 
21 


i 
14 


In relation to other courses, the workload in this course 
was much heavier. 

There were too many demands placed on the student. 

The material presented in this course was much too difficult. 
The pace of this course was too fast. 

I have spent considerably more time in this course than in 
others. 

The grading system was quite fair. 

The tests were too difficult, 


Factor II (Instructor Effectiveness): 


Factor Loading 


0.82 
0.80 
0.61 


0.60 


Items 


(17) 
( 5) 
(44) 


( 8) 


The instructor was quite helpful. 

The instructor made himself readily available. 

The instructor had a tremendous knowledge of the subject 
matter. 

The instructor had a good understanding of the subject matter. 


Factor III (Attitude Toward Course and Instructional Method): 


Factor Loading 


0.79 
0.75 
0.65 
0.65 
0.59 
0,55 


Items 


30 
3 
4 

(28) 


36 
29 


If another course were taught in this way, ! would take it. 


23. 


I was enthusiastic and receptive to this method of instruction. 


I would recommend this course to a friend. 

I was highly motivated by this method of instruction. 
When other methods of instruction are used, I learn more. 
My overall evaluation of this course is excellent. 


Factor IV (Relevancy of Course and Instructional Method): 


Factor Loading 
0.66 


0.55 
0.53 
0.50 


Items 
(26) 
(39) 
(38) 
(27) 


a course caused me to make more effective use of my study 
time. 

This method of instruction taught me to pursue the subject 

on my own. 

This course taught me new ways to understand and evaluate 


problems. 
The amount of useful material learned in this course was 


negligible. 
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Table 10. Analysis of variance results for four subscales of the course evaluation in ten civil engineering technology courses. 


COURSE 


Highway Design 
and Construction 


Surveying II 


tatics 


Surveying I 


Structural Steel 


Strength of 
Materials 


Hydraulics 


Cost Estimating 


Structural Concrete 


cc 1 Instructor Effectiveness | 


cece : ile oe rey || 
4.4 o 01 
(47) 1(1,52) an Uap) 
tay. nes.) 31. 48 10.36 N.S. 
38) (1,48) (16) 
ia 2.9 
(56) (56) 
<.01 Te 3,5 


Attitude Toward Course Relevancy of Course and 
and Instructional Method Instructional Method 


|e De [re 
1.33 N.S. 0.02. Bs, 
it, an (47) (1.52) 


i 1. 5) 
| si 


5.5 3.3 12.92 
(14) | (28) |(1,39) 


3.4 11.6 [18.65 <,01 
(32) | (39) |(1,64) 
3.1 ae 18 N.S. 
(7) (i 2) (1,17) 


53 | 2a 8.00 <0f 

(12) | (18) |(1,28) (18) | (1,28) 
4.3 

(16) 


3.7 
(38) 


NUTE: 

Xe = mean of control group Total points possible for each subscale: 

= tI Course Workload and Grading....... eiireemreaihemiiseaell points 
Xe = mean of experimental group 2) Instructor Effectiveness.. ‘aise diese wie’ POINGS 
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(3) Attitude Toward Course and Instru setional Method... .6 points 
(4) Relevancy of Course ard Instructional Method...... 4 points 
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Summary 


The evaluation was based on a pretest/posttest design to compare two operationally 
defined instructional strategies: an individually-paced method and a lecture-based 
method. The effectiveness of the individually-paced format was determined primarily 
through the analysis of six evaluation problem statements. Due to the limited time 
allotted for the evaluation and the relatively small number of classes investigated 
within each course, the results must be viewed as tentative. 


In terms of cognitive achievement, the individually-paced students did relatively well. 
Although the results were not dramatic, the individually-paced students generally out- 
performed the lecture-based students in terms of student exit behaviors (as measured 

by the Summative Posttests); however, the range of scores was slightly greater for 
individually-paced students than for lecture-based students. More variability in terms 
of student exit behaviors was not anticipated in a learning for mastery strategy. The 
experimental group also scored higher than the control group on a ten-week retention 
test in Structural Steel, although the results were not statistically significant. 


In contrast to the results found in cognitive achievement, individually-paced students 
did not appear to display a more positive affect toward subject matter than lecture- 
based students. In fact, four empirically validated subscales (Course Workload and 
Grading, Instructor Effectiveness, Attitude Toward Course and Instructional Method, 
and Relevancy of Course and Instructional Method) were rated higher by lectu »-based 
students in the majority of courses. This apparent discrepancy between the cognitive 
and affective domains may be related to the fact that the individually-paced instruc- 
tional method was totally new to the students and that the students were required to 
become more active participants in the learning process. 


Aside from affective and cognitive characteristics, the topics of procrastination and 
final grade distribution were investigated. In the evaluation, the definition of a 
procrastinator was expanded to include any student who dropped a course, received an 
incomplete, or failed a course. Using this definition, a chi-square analysis indicated 
there was no observed tendency for students to procrastinate more in individually-paced 
courses than in lecture-based courses. \'ith respect to final grades assigned to 
students, a chi-square analysis in one course (Structural Concrete) revealed there was 
no reliable relationship observed between final grades assigned and instructional method 
employed. This apparent independent relationship was interesting in light of the fact 
that statistically significant differences favoring the experimental group were found 
on the Structural Concrete Summative Posttest. * 


The adequacy of the materials used in the individually-paced format was also investi- 
gated. When the individually-paced materials were used in an institution outside the 
state of South Carolina, the results from one course (Surveying II) tended to support 
the idea that these materials could be used effectively elsewhere. 


There were indications, however, that individually-paced students would have pre- 

ferred learning the material (e.g., modules and media presentations) in a different 
manner. These indications were born out of information gathered from the Course Eval- 
uations, Course Questionnaires and Instructor Questionnaires. Results from these 
measuring instruments indicated that students would have preferred more instructor input 
as well as a more rigidly defined course time-table. Students also complained that 
there were too many students per instructor in the individually-paced courses. On the 
other hand, instructors stressed the importance of the individually-paced method in 
terms of student ability and motivation. Viewing the disadvantages of the individually- 
paced method, the instructor-student ratio appeared to be the critical feature. If the 
ratio is reasonably low (e.g., one instructor for every ten students), the disadvantages 
of the individually-paced method appear to be minimal. 31 


